Optimal function approximation with ReLU neural networks

In this paper, we consider the optimal approximations of univariate functions with feed-forward ReLU neural networks. We attempt to answer following questions. For given function and network, what is minimal possible approximation error? How fast does error decrease network size? Is attainable by current training techniques? Theoretically, introduce necessary sufficient conditions for convex functions. give lower upper bounds errors, rate that measures how decreases size. architectures are presented generate approximations. then propose an algorithm compute prove its convergence. conduct experiments validate effectiveness compare other approaches. also demonstrate theoretical limit errors not attained networks trained stochastic gradient descent optimization, which indicates expressive power has been exploited full potential.